Abstract

BACKGROUND: Health-evidence.ca is an online registry of systematic reviews evaluating the effectiveness of public health interventions. Extensive searching of bibliographic databases is required to keep the registry up to date. However, search filters have been developed to assist in searching the extensive amount of published literature indexed. Search filters can be designed to find literature related to a certain subject (i.e. content-specific filter) or particular study designs (i.e. methodological filter). The objective of this paper is to describe the development and validation of the health-evidence.ca Systematic Review search filter and to compare its performance to other available systematic review filters. METHODS: This analysis of search filters was conducted in MEDLINE, EMBASE, and CINAHL. The performance of thirty-one search filters in total was assessed. A validation data set of 219 articles indexed between January 2004 and December 2005 was used to evaluate performance on sensitivity, specificity, precision and the number needed to read for each filter. RESULTS: Nineteen of 31 search filters were effective in retrieving a high level of relevant articles (sensitivity scores greater than 85%). The majority achieved a high degree of sensitivity at the expense of precision and yielded large result sets. The main advantage of the health-evidence.ca Systematic Review search filter in comparison to the other filters was that it maintained the same level of sensitivity while reducing the number of articles that needed to be screened. CONCLUSIONS: The health-evidence.ca Systematic Review search filter is a useful tool for identifying published systematic reviews, with further screening to identify those evaluating the effectiveness of public health interventions. The filter that narrows the focus saves considerable time and resources during updates of this online resource, without sacrificing sensitivity.
Hide

Abstract

OBJECTIVES: Systematic reviews of the literature are instrumental for bridging research to health care practice and are widely available through databases such as MEDLINE and EMBASE. Search strategies have been developed to aid users in MEDLINE, but no empirical work has been done for EMBASE. The objective of this study was to develop search strategies that optimize the retrieval of methodologically sound systematic reviews from EMBASE. STUDY DESIGN AND SETTING: An analytic survey was conducted, comparing hand searches of 55 journals with retrievals from EMBASE for 4,843 candidate search terms and 17,004 combinations. Candidate search strategies were run in EMBASE, the retrievals being compared with the hand search data. The sensitivity, specificity, precision, and accuracy of the search strategies were calculated. RESULTS: Two hundred twenty (16.2%) of the 1,354 articles classified as a review met basic criteria for scientific merit. Combinations of search terms reached peak sensitivities of 94.6% with specificity at 63.7%, whereas combinations of search terms to optimize specificity reached peak specificities of 99.3% with sensitivity at 61.4%. CONCLUSION: Empirically derived search strategies can achieve high sensitivity and specificity for retrieving methodologically sound systematic reviews from EMBASE.
Hide

Abstract

Search filters are increasingly used to search medical databases to identify specific topics or study designs. In particular, search filters have been designed to help health-care professionals identify systematic reviews of the effectiveness of health interventions. Identifying systematic reviews in databases such as MEDLINE is problematic and research has previously been undertaken into methods to design search filters that retrieve systematic reviews effectively. The aim of this study was to improve previously developed methods to derive a more objective search strategy to identify systematic reviews in MEDLINE. A 'quasi-gold standard' collection of known systematic reviews was identified. A frequency analysis of words within a subset of the 'quasi-gold standard' was undertaken followed by a statistical analysis of the most frequently occurring words. This analysis determined which terms would best distinguish between systematic reviews, non-systematic reviews and non-reviews. The performance of the best models was tested on the remaining subset of 'quasi-gold standard' records and then using the OVID interface to MEDLINE. The best model had a sensitivity of 73.4% for systematic reviews in the test set and 84.2% when used with the validation set. The best model had a specificity of 98.3% in the test set and 93% in the validation set. When tested on the same 'quasi-gold standard' using OVID MED-LINE the model showed 100% sensitivity and 4.4% precision. The number of times a term occurs in a record adds discriminatory power to the search strategy. Apparently highly relevant terms chosen subjectively do not perform as well as those derived by a statistical approach. Some search terms may not immediately seem useful in identifying systematic reviews, but when used in combination with other terms they prove to be highly discriminating. The best performing filters were tested on the OVID interface, but without frequency and term weightings. Their performance was also compared to previously published filters. One of the strategies was found to perform better with respect to sensitivity than previously published filters, although with lower precision.
Hide

Abstract

OBJECTIVE: Advances from health research are not well applied giving rise to over- and underuse of resources and inferior care. Knowledge translation (KT), actions and processes of getting research findings used in practice, can improve research application. The KT literature is difficult to find because of nonstandardized terminology, rapid evolution of the field, and it is spread across several domains. We created multiple search filters to retrieve KT articles from MEDLINE. STUDY DESIGN AND SETTING: Analytic survey using articles from 12 journals tagged as having KT content and also as describing a KT application or containing a KT theory. RESULTS: Of 2,594 articles, 579 were KT articles of which 201 were about KT applications and 152 about KT theory. Search filter sensitivity (retrieval efficiency) maximized at 83%-94% with specificity (no retrieval of irrelevant material) approximately 50%. Filter performances were enhanced with multiple terms, but these filters often had reduced specificity. Performance was higher for KT applications and KT theory articles. These filters can select KT material although many irrelevant articles also will be retrieved. CONCLUSION:
Hide

Abstract

Background: Up to now no published methodological filter designed for the retrieval of diagnostic test accuracy (DTA) studies has proved sensitive enough for use within a Cochrane DTA systematic review. A recent study tested the hypothesis that normal cross-sectional studies should be treated differently from delayed cross-sectional studies (longitudinal analyses). Objectives: i) further test the hypothesis that normal cross-sectional studies and delayed cross-sectional studies are generally described differently in the published literature, ii) refine an unpublished methodological filter designed for identification of delayed cross-sectional studies (this will be done in MEDLINE (Ovid)), and iii) develop an equivalent filter for use in PubMed. Methods: An unpublished filter will be further validated by expansion of the data set. Key elements of reports of potential studies for inclusion within Cochrane DTA reviews that focus on longitudinal diagnosis and prediction will be entered into textual analysis software so as to refine the existing unpublished filter. The filter will then be further tested by a large dataset of reports of potentially relevant studies from existing literature on methodological filters. Finally, a PubMed equivalent filter will be developed. Results: The results will show the new filter's sensitivity, specificity, precision and accuracy and will be presented at the Cochrane Colloquium in Madrid, October 2011. Conclusions: DTA studies generally fall into two camps - normal cross-sectional studies (e.g. looking at the accuracy of a new test to see if someone is pregnant or not) and delayed cross-sectional studies (e.g. a new test to see whether someone will develop symptomatic dementia from a 'pre-dementia' state). The filters published so far treat these studies as one. The literature therefore condemns these filters based on potentially invalid criteria.
Hide

Abstract

BACKGROUND: Cluster randomized trials (CRTs) present unique methodological and ethical challenges. Researchers conducting systematic reviews of CRTs (e.g., addressing methodological or ethical issues) require efficient electronic search strategies (filters or hedges) to identify trials in electronic databases such as MEDLINE. According to the CONSORT statement extension to CRTs, the clustered design should be clearly identified in titles or abstracts; however, variability in terminology may make electronic identification challenging. Our objectives were to (a) evaluate sensitivity ("recall") and precision of a well-known electronic search strategy ("randomized controlled trial" as publication type) with respect to identifying CRTs, (b) evaluate the feasibility of new search strategies targeted specifically at CRTs, and (c) determine whether CRTs are appropriately identified in titles or abstracts of reports and whether there has been improvement over time. METHODS: We manually examined a wide range of health journals to identify a gold standard set of CRTs. Search strategies were evaluated against the gold standard set, as well as an independent set of CRTs included in previous systematic reviews. RESULTS: The existing strategy (randomized controlled trial.pt) is sensitive (93.8%) for identifying CRTs, but has relatively low precision (9%, number needed to read 11); the number needed to read can be halved to 5 (precision 18.4%) by combining with cluster design-related terms using the Boolean operator AND; combining with the Boolean operator OR maximizes sensitivity (99.4%) but would require 28.6 citations read to identify one CRT. Only about 50% of CRTs are clearly identified as cluster randomized in titles or abstracts; approximately 25% can be identified based on the reported units of randomization but are not amenable to electronic searching; the remaining 25% cannot be identified except through manual inspection of the full-text article. The proportion of trials clearly identified has increased from 28% between the years 2000-2003, to 60% between 2004-2007 (absolute increase 32%, 95% CI 17 to 47%). CONCLUSIONS: CRTs should include the phrase "cluster randomized trial" in titles or abstracts; this will facilitate more accurate indexing of the publication type by reviewers at the National Library of Medicine, and efficient textword retrieval of the subset employing cluster randomization.
Hide

Abstract

OBJECTIVES: For the measurement of patient-reported outcomes, such as (health-related) quality of life, often many measurement instruments exist that intend to measure the same construct. To facilitate instrument selection, our aim was to develop a highly sensitive search filter for finding studies on measurement properties of measurement instruments in PubMed and a more precise search filter that needs less abstracts to be screened, but at a higher risk of missing relevant studies. METHODS: A random sample of 10,000 PubMed records (01-01-1990 to 31-12-2006) was used as a gold standard. Studies on measurement properties were identified using an exclusion filter and hand searching. Search terms were selected from the relevant records in the gold standard as well as from 100 systematic reviews of measurement properties and combined based on sensitivity and precision. The performance of the filters was tested in the gold standard as well as in two validation sets, by calculating sensitivity, precision, specificity, and number needed to read. RESULTS: We identified 116 studies on measurement properties in the gold standard. The sensitive search filter was able to retrieve 113 of these 116 studies (sensitivity 97.4%, precision 4.4%). The precise search filter had a sensitivity of 93.1% and a precision of 9.4%. Both filters performed very well in the validation sets. CONCLUSION: The use of these search filters will contribute to evidence-based selection of measurement instruments in all medical fields.
Hide